Image sensing apparatus

ABSTRACT

An image sensing apparatus has an image sensing device. The image sensing device includes a photoelectric conversion element that photoelectrically converts an optical image to acquire image data, and a readout control unit that reads out, in accordance with a supplied readout rule, the image data acquired by the photoelectric conversion element. The image sensing device also includes an image scaling ratio selection unit that selects the scaling ratio of the image to be output, a readout scheme selection unit that selects, in accordance with the selected image scaling ratio, the readout scheme of the image data to be read out from the photoelectric conversion element by the readout control unit, and a readout rule supply unit that supplies, to the readout control unit, a readout rule corresponding to the readout scheme selected by the readout scheme selection unit.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a Continuation Application of PCT Application No.PCT/JP2004/000575, filed Jan. 23, 2004, which was published under PCTArticle 21(2) in Japanese.

This application is based upon and claims the benefit of priority fromprior Japanese Patent Application No. 2003-020717, filed Jan. 29, 2003,the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a digital image sensing apparatus and,more particularly, to a digital image sensing apparatus that quicklygenerates a high-quality image with a smaller number of pixels than thatof an image sensing device mounted in the image sensing apparatus.

2. Description of the Related Art

Along with the recent rapid spread of personal computers, the demand fordigital cameras serving as image input devices is increasing. Inaddition, high-quality recording apparatuses such as digital videorecorders are widely used as moving image recorders.

The image quality of an electronic still camera depends on severalfactors. Especially, the number of pixels of an image sensing element isa very important factor for the resolution of a sensed image. Somerecent commercially available electronic still cameras have more than 5million pixels. However, data of 5 million pixels is not alwaysnecessary for all application purposes. Images displayed on the Webs ofthe Internet often have, if anything, smaller pixel sizes.

In current digital cameras, the transfer time from the image sensingelement to the image memory is a bottleneck. Most models having a largenumber of pixels cannot execute high-speed continuous shooting. Inaddition, even digital cameras need to have a moving image sensingfunction as an additional function. Hence, transfer to the memory mustbe done at a high speed. For this purpose, the data amount to beprocessed is preferably reduced in advance.

When the number of pixels of an output image is smaller than that of theimage sensing element, the number of pixels to be used is limited inadvance. Alternatively, a plurality of pixels are averaged and read outby one clock. With this processing, the amount of data to be transferredfrom the image sensing element to the memory can be reduced, and thememory transfer rate can be increased.

In size reduction by linear interpolation, an image having a large sizeis generated by using all pixels. Then, an image with a small size isgenerated by linear interpolation.

Such resizing by linear interpolation can ensure a high image quality.However, since linear interpolation is executed by using all pixel data,the arithmetic amount is large. Hence, this method is inappropriate forthe above-described continuous shooting function or moving imagesensing.

There is a method of reducing the data amount of memory readout, inwhich an integration function is added to the image sensing element sothat a reduced image is generated by reading out a small number ofaveraged data. Jpn. Pat. Appln. KOKAI Publication No. 2001-245141discloses a high-speed image reducing method using this method.

Jpn. Pat. Appln. KOKAI Publication No. 2001-016441 discloses anapparatus that executes data thinning and also corrects distortion ofdata when the number of resolutions is limited. An embodiment of thisreference discloses creation of 400-dpi data by an apparatus having aresolution of 600 dpi. When 600-dpi data is directly thinned out, thedata is distorted. To cope with this, pixel data to compensate for thedistortion of positions is generated from the 600-dpi data by linearinterpolation. The apparatus disclosed in Jpn. Pat. Appln. KOKAIPublication No. 2001-016441 creates 400-dpi data by executinginterpolation totally using all 600-dpi data obtained by scanning.

BRIEF SUMMARY OF THE INVENTION

The present invention is directed to an image sensing apparatus thatshortens the time required to read out image data from an image sensingelement and can form a high-resolution image with few distortion in awide scaling ratio range.

An image sensing apparatus according to the present invention has animage sensing device. The image sensing device includes a photoelectricconversion element that photoelectrically converts an optical image toacquire image data, and a readout control unit that reads out, inaccordance with a supplied readout rule, the image data acquired by thephotoelectric conversion element. The image sensing device also includesan image scaling ratio selection unit that selects the scaling ratio ofthe image to be output, a readout scheme selection unit that selects, inaccordance with the selected image scaling ratio, the readout scheme ofthe image data to be read out from the photoelectric conversion elementby the readout control unit, and a readout rule supply unit thatsupplies, to the readout control unit, a readout rule corresponding tothe readout scheme selected by the readout scheme selection unit.

Advantages of the invention will be set forth in the description whichfollows, and in part will be obvious from the description, or may belearned by practice of the invention. Advantages of the invention may berealized and obtained by means of the instrumentalities and combinationsparticularly pointed out hereinafter.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate presently preferred embodiments ofthe invention, and together with the general description given above andthe detailed description of the preferred embodiments given below, serveto explain the principles of the invention.

FIG. 1 shows the arrangement of an image sensing apparatus according tothe first embodiment of the present invention.

FIG. 2 shows an example in which an image having an RG/GB Bayer matrixis subjected to 14/16 reduction conversion in the vertical direction.

FIG. 3 shows a state in which, of 16 pixels before conversion, theeighth and ninth pixel data from the upper side are omitted in theconversion shown in FIG. 2.

FIG. 4 shows an example in which two of 16 pixels are thinned out inboth the horizontal and vertical directions.

FIG. 5 shows distortion correction conversion of the data of the firstcolumn on the left side of the pixel data that is thinned out accordingto the example shown in FIG. 4.

FIG. 6 shows an example in which two of eight pixels are thinned out inboth the horizontal and vertical directions.

FIG. 7 shows the arrangement of a filter processing unit for aphotoelectric conversion element including a single-chip color imagesensing element.

FIG. 8 shows the arrangement of a filter processing unit for aphotoelectric conversion element including a monochrome image sensingelement or a multiple-chip color image sensing element.

FIG. 9 shows size change corresponding to thinning readout+distortioncorrection processing.

FIG. 10 shows size change corresponding to averaging readout+linearinterpolation processing.

FIG. 11 shows readout scheme switching and size change corresponding toan image scaling ratio in the image sensing apparatus shown in FIG. 1.

FIG. 12 shows the arrangement of an image sensing apparatus according tothe second embodiment of the present invention.

FIG. 13 schematically shows pixel data of two fields that are adjacenttime-serially to form one frame, the pixel data being read out by ahorizontal averaging vertical interlaced readout.

FIG. 14 schematically shows pixel data in the same area of two framesthat are adjacent time-serially, the pixel data being read out bythinning readout and readout area reference position shift processing.

FIG. 15 shows pixel position of frame readout in a readout by repeating6/8 thinning readout, in which the readout start position of the readoutarea matches the upper left pixel of the pixel matrix of a photoelectricconversion element.

FIG. 16 shows pixel position of frame readout in a readout by repeating6/8 thinning readout, in which the readout end position of the readoutarea matches the lower right pixel of the pixel matrix of thephotoelectric conversion element.

FIG. 17 shows the arrangement of an image sensing apparatus according tothe third embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The embodiments of the present invention will be described below withreference to the accompanying drawings.

First Embodiment

FIG. 1 shows the arrangement of an image sensing apparatus according tothe first embodiment of the present invention. An image sensingapparatus 100 has an imaging optical system 110 that forms an opticalimage of an object and an image sensing device 120 that outputs an imagesignal in a predetermined region of the optical image formed by theimaging optical system 110. The image sensing device 120 has atwo-dimensional photoelectric conversion element (image sensing element)122 that photoelectrically converts the optical image formed by theimaging optical system 110 to acquire image data (pixel data set) and areadout control unit 124 that reads out the image data acquired by thephotoelectric conversion element 122 in accordance with a suppliedreadout rule.

The image sensing apparatus 100 also has an image scaling ratioselection unit 132 that selects the scaling ratio of the image to beoutput, a readout scheme selection unit 134 that selects the readoutscheme of the image data to be read out from the photoelectricconversion element 122 by the readout control unit 124 in accordancewith the selected image scaling ratio, and a readout rule supply unit140 that supplies a readout rule corresponding to the readout schemeselected by the readout scheme selection unit 134 to the image sensingdevice 120.

The readout scheme selection unit 134 selects one readout scheme of athinning readout mode, an averaging readout mode, and a full-pixelreadout mode in accordance with the selected image scaling ratio. Thereadout rule supply unit 140 includes a thinning readout rule settingunit 142 that sets a readout rule corresponding to the thinning readoutmode, averaging readout rule setting unit 144 that sets a readout rulecorresponding to the averaging readout mode, and full-pixel readout rulesetting unit 146 that sets a readout rule corresponding to thefull-pixel readout mode.

The readout rule supply unit 140 selectively operates one of thethinning readout rule setting unit 142, averaging readout rule settingunit 144, and full-pixel readout rule setting unit 146, whichcorresponds to the readout scheme selected by the readout schemeselection unit 134. Then, a readout rule corresponding to the readoutscheme selected by the readout scheme selection unit 134 is supplied tothe readout control unit 124; the readout control unit 124 reads outpixel data from the photoelectric conversion element 122 in accordancewith the readout rule supplied from the readout rule supply unit 140.

More specifically, for an image scaling ratio lower than 100%, i.e., forimage reduction, the readout scheme selection unit 134 selects one ofthe thinning readout mode and averaging readout mode. For an imagescaling ratio of 100% or more, i.e., for image enlargement, the readoutscheme selection unit 134 selects the full-pixel readout mode. The imagescaling ratio indicates the ratio of the size (the number of pixels inthe horizontal and vertical area) of two-dimensionally arrayed pixeldata of an image to be output (e.g., displayed on an image display unit174) to the region (the number of pixels in the horizontal and verticalarea) of two-dimensionally arrayed pixel data acquired by thephotoelectric conversion element 122.

For an image scaling ratio lower than 100%, the readout scheme selectionunit 134 selects one of the thinning readout mode and averaging readoutmode on the basis of the image scaling ratio and an important one offactors (contrast, resolution, distortion, luminance moiré, and colormoiré) of the quality of the image to be output. In other words, for animage scaling ratio lower than 100%, the readout scheme selection unit134 selects one of the thinning readout mode and averaging readout modeon the basis of the image scaling ratio and assumed object.

The image sensing apparatus 100 also has a distortion correction unit150 that executes distortion correction for the image signal output fromthe image sensing device 120 in the thinning readout mode, a linearinterpolation size change unit 164 that executes size change by linearinterpolation for the image signal output from the image sensing device120 in the averaging readout mode and full-pixel readout mode, and aselector 162 that selectively sends the image signal output from theimage sensing device 120 to one of the distortion correction unit 150and linear interpolation size change unit 164 in accordance with thereadout scheme selected by the readout scheme selection unit 134.

When the readout scheme selection unit 134 selects the thinning readoutmode, the selector 162 sends the image signal from the image sensingdevice 120 to the distortion correction unit 150. When the readoutscheme selection unit 134 selects the averaging readout mode orfull-pixel readout mode, the selector 162 sends the image signal fromthe image sensing device 120 to the linear interpolation size changeunit 164.

The distortion correction unit 150 has a filter processing unit 152 thatexecutes filter processing for the image signal from the image sensingdevice 120 and a filter coefficient setting unit 154 that sets a filtercoefficient to be used for the filter processing by the filterprocessing unit 152 in accordance with the readout rule set by thethinning readout rule setting unit 142.

The filter coefficient setting unit 154 has an LUT storage unit 156 thatstores a lookup table (LUT) containing a plurality of filtercoefficients and a filter coefficient selection unit 158 that selects afilter coefficient from the lookup table stored in the LUT storage unit156.

The filter coefficient setting unit 154 need not always have the LUTstorage unit 156 and filter coefficient selection unit 158. The filtercoefficient setting unit 154 may calculates a filter coefficient byarithmetic processing corresponding to the readout rule set by thethinning readout rule setting unit 142.

The filter coefficient setting unit 154 that uses an LUT requires alarge memory capacity to store the LUT, though the load of arithmeticprocessing can be small. On the other hand, the filter coefficientsetting unit 154 that uses no LUT requires no large memory capacity,though the load of arithmetic processing is large.

The image sensing apparatus 100 also has an image signal processing unit172 that executes predetermined processing (e.g., white balance, graylevel conversion, or edge enhancement) for the image signal output fromthe distortion correction unit 150 in the thinning readout mode or theimage signal output from the linear interpolation size change unit 164in the averaging readout mode or full-pixel readout mode, an imagedisplay unit 174 that displays an image in accordance with the imagesignal output from the image signal processing unit 172, and an imagerecording unit 176 that records an image in accordance with the imagesignal output from the image signal processing unit 172.

The image sensing device 120 can execute a thinning readout operation.By the thinning readout operation, the image sensing device 120 can readout pixels in a specific region on the photoelectric conversion element122 in a shorter time than the readout of all pixels.

For example, when the photoelectric conversion element 122 is an imagesensing element using CMOS, the image sensing device 120 can designate areadout position by using shift registers in both the horizontal andvertical directions.

More specifically, assuming that the ith element of the jth line isdefined as C(i, j) and pixels from there in the horizontal direction aredefined as C(i+1, j), C(i+2, j), C(i+3, j), C(i+4, j), C(i+5, j), C(i+6,j), C(i+7, j), C(i+8, j), . . . , it can read out pixels thinning outthem at arbitrary horizontal positions, like C(i+1, j), C(i+2, j),C(i+3, j), C(i+4, j), C(i+7, j), C(i+8, j), . . . for example.

This also applies to the vertical direction; for pixels arrayed in thedirection of lines, e.g., jth line, (j+1)th line, (j+2)th line . . . ,it can read out pixels thinning out them at arbitrary lines.

When the photoelectric conversion element 122 is a CCD, since it readsout data while shifting charges in the horizontal direction, the imagesensing device 120 reads out all pixels in the horizontal direction butcan read out pixels thinning out them in the vertical direction.

The distortion correction unit 150 interpolates the thinned digitalimage data with omitted information and also executes filter processingfor magnification conversion. That is, in this specification, distortioncorrection means simultaneously executing “interpolation” and“magnification conversion”.

In bilinear interpolation, when the magnification conversion is limitedto a rational number (integral ratio), and linear interpolation isrepeated twice, the algorithm is simplified. FIG. 2 shows an example inwhich an image having an RG/GB Bayer matrix is subjected to 14/16reduction conversion in the horizontal direction. Referring to FIG. 2,the upper stage indicates a one-dimensional data array of pixels beforereduction conversion, and the lower stage indicates a one-dimensionaldata array of pixels after reduction conversion.

This conversion can be expressed by a matrix given by $\begin{matrix}{\begin{pmatrix}{Rc}_{0} \\{Gc}_{1} \\{Rc}_{2} \\{Gc}_{3} \\{Rc}_{4} \\{Gc}_{5} \\{Rc}_{6} \\{Gc}_{7} \\{Rc}_{8} \\{Gc}_{9} \\{Rc}_{10} \\{Gc}_{11} \\{Rc}_{12} \\{Gc}_{13}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & \frac{13}{14} & 0 & \frac{1}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & \frac{12}{14} & 0 & \frac{2}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & \frac{11}{14} & 0 & \frac{3}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & \frac{10}{14} & 0 & \frac{4}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & \frac{9}{14} & 0 & \frac{5}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & \frac{8}{14} & 0 & \frac{6}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{7}{14} & 0 & \frac{7}{14} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{6}{14} & 0 & \frac{8}{14} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{5}{14} & 0 & \frac{9}{14} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{4}{14} & 0 & \frac{10}{14} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{3}{14} & 0 & \frac{11}{14} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{2}{14} & 0 & \frac{12}{14} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{1}{14} & 0 & \frac{13}{14}\end{pmatrix}{\begin{pmatrix}{Ri}_{0} \\{Gi}_{1} \\{Ri}_{2} \\{Gi}_{3} \\{Ri}_{4} \\{Gi}_{5} \\{Ri}_{6} \\{Gi}_{7} \\{Ri}_{8} \\{Gi}_{9} \\{Ri}_{10} \\{Gi}_{11} \\{Ri}_{12} \\{Gi}_{13} \\{Ri}_{14} \\{Gi}_{15}\end{pmatrix}.}}} & (1)\end{matrix}$

In Equation (1), Ri_(2p) and Gi_(2p+1) (p is an integer that is nosmaller than 0 and smaller than 7) represent pixel data of pixelscontinuously arrayed in the horizontal direction in the photoelectricconversion element 122 and have consecutive subscripts corresponding tothe positions of the pixels arrayed in the horizontal direction. Rc_(2q)and Gc_(2q+1) (q is an integer that is no smaller than 0 and smallerthan 6) represent pixel data after conversion and have consecutivesubscripts corresponding to the positions of the pixels arrayed in thehorizontal direction.

For example, using Ri₂ and Ri₄, Rc₂ after conversion is given byRc ₂= 12/14Ri ₂₂+ 2/14Ri ₄.  (2)

Equation (1) generally expresses the conversion from 16 pixels to 14pixels, in which each pixel is converted in the above-described manner.

FIG. 3 shows a state in which, of the 16 pixels before conversion, theeighth and ninth pixel data from the left side are omitted in theconversion shown in FIG. 2. In this case, omitted pixel data Ri₈ and Gi₉are preferably linearly interpolated by using close pixel data (Ri₆ andRi₁₀ for Ri₈ and Gi₇ and Gi₁₁ for Ri₉) of the same channel in accordancewith $\begin{matrix}{{{Ri}_{8} = \frac{{Ri}_{6} + {Ri}_{10}}{2}},{{Gi}_{9} = {\frac{{Gi}_{7} + {Gi}_{11}}{2}.}}} & (3)\end{matrix}$

When Ri₈ and Gig in Equation (1) are replaced in accordance withEquation (3), we obtain $\begin{matrix}{\begin{pmatrix}{Rc}_{0} \\{Gc}_{1} \\{Rc}_{2} \\{Gc}_{3} \\{Rc}_{4} \\{Gc}_{5} \\{Rc}_{6} \\{Gc}_{7} \\{Rc}_{8} \\{Gc}_{9} \\{Rc}_{10} \\{Gc}_{11} \\{Rc}_{12} \\{Gc}_{13}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & \frac{13}{14} & 0 & \frac{1}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & \frac{12}{14} & 0 & \frac{2}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & \frac{11}{14} & 0 & \frac{3}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & \frac{10}{14} & 0 & \frac{4}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & \frac{9}{14} & 0 & \frac{5}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & \frac{8}{14} & 0 & \frac{6}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{7}{14} & 0 & \frac{7}{14} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{6}{14} & 0 & \frac{8}{14} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{5}{14} & 0 & \frac{9}{14} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{4}{14} & 0 & \frac{10}{14} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{3}{14} & 0 & \frac{11}{14} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{2}{14} & 0 & \frac{12}{14} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{1}{14} & 0 & \frac{13}{14}\end{pmatrix}{\begin{pmatrix}{Ri}_{0} \\{Gi}_{1} \\{Ri}_{2} \\{Gi}_{3} \\{Ri}_{4} \\{Gi}_{5} \\{Ri}_{6} \\{Gi}_{7} \\{{Ri}_{6} + {Ri}_{10}} \\2 \\{{Gi}_{7} + {Gi}_{11}} \\2 \\{Ri}_{10} \\{Gi}_{11} \\{Ri}_{12} \\{Gi}_{13} \\{Ri}_{14} \\{Gi}_{15}\end{pmatrix}.}}} & (4)\end{matrix}$

Sixteen arrays Ri₀, Gi₁, . . . , Ri₁₄, and Gi₁₅ on the right-hand sideof the Equation (4) can be expressed by $\begin{matrix}{\begin{pmatrix}{Ri}_{0} \\{Gi}_{1} \\{Ri}_{2} \\{Gi}_{3} \\{Ri}_{4} \\{Gi}_{5} \\{Ri}_{6} \\{Gi}_{7} \\{{Ri}_{6} + {Ri}_{10}} \\2 \\{{Gi}_{7} + {Gi}_{11}} \\2 \\{Ri}_{10} \\{Gi}_{11} \\{Ri}_{12} \\{Gi}_{13} \\{Ri}_{14} \\{Gi}_{15}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & \frac{1}{2} & 0 & \frac{1}{2} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{1}{2} & 0 & \frac{1}{2} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 1\end{pmatrix}\begin{pmatrix}{Ri}_{0} \\{Gi}_{1} \\{Ri}_{2} \\{Gi}_{3} \\{Ri}_{4} \\{Gi}_{5} \\{Ri}_{6} \\{Gi}_{7} \\{Ri}_{10} \\{Gi}_{11} \\{Ri}_{12} \\{Gi}_{13} \\{Ri}_{14} \\{Gi}_{15}\end{pmatrix}}} & (5)\end{matrix}$When this is substituted into Equation (4), and the product of thematrix is calculated, we obtain $\begin{matrix}{\begin{pmatrix}{Rc}_{0} \\{Gc}_{1} \\{Rc}_{2} \\{Gc}_{3} \\{Rc}_{4} \\{Gc}_{5} \\{Rc}_{6} \\{Gc}_{7} \\{Rc}_{8} \\{Gc}_{9} \\{Rc}_{10} \\{Gc}_{11} \\{Rc}_{12} \\{Gc}_{13}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & \frac{13}{14} & 0 & \frac{1}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & \frac{12}{14} & 0 & \frac{2}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & \frac{11}{14} & 0 & \frac{3}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & \frac{10}{14} & 0 & \frac{4}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & \frac{9}{14} & 0 & \frac{5}{14} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & \frac{11}{14} & 0 & \frac{3}{14} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{21}{28} & 0 & \frac{7}{28} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & \frac{3}{14} & 0 & \frac{11}{14} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{5}{28} & 0 & \frac{23}{28} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{4}{14} & 0 & \frac{10}{14} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{3}{14} & 0 & \frac{11}{14} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{2}{14} & 0 & \frac{12}{14} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{1}{14} & 0 & \frac{13}{14}\end{pmatrix}{\begin{pmatrix}{Ri}_{0} \\{Gi}_{1} \\{Ri}_{2} \\{Gi}_{3} \\{Ri}_{4} \\{Gi}_{5} \\{Ri}_{6} \\{Gi}_{7} \\{Ri}_{10} \\{Gi}_{11} \\{Ri}_{12} \\{Gi}_{13} \\{Ri}_{14} \\{Gi}_{15}\end{pmatrix}.}}} & (6)\end{matrix}$

This conversion is equivalent to Equation (4) and has 14 pixel dataoutputs corresponding to 14 pixel data inputs. In other words, thisconversion obtains 14 pixel data after 14/16 reduction conversion fromthe 14 pixel data except the pixel data Ri₈ and Gi₉.

FIG. 4 shows an example in which two of 16 pixels are thinned out inboth the horizontal and vertical directions. In this example, the eighthand ninth pixels are thinned out in both the horizontal and verticaldirections.

FIG. 5 shows conversion of the first column on the left side of thepixel data that is read out with thinning-out according to the exampleshown in FIG. 4. As shown in FIG. 5, actually readout pixel data are 14data, Ri₀, Gi₁, Ri₂, Gi₃, Ri₄, Gi₅, Ri₆, Gi₇, Ri₁₀, Gi₁₁, Ri₁₂, Gi₁₃,Ri₁₄, and Gi₁₅, in the vertical direction.

Equation (6) is equivalent to conversion that obtains 14 pixel dataafter 14/16 reduction conversion from the 14 pixel data except thethinned eighth (eighth row) and ninth (ninth row) pixel data.

As is apparent from the equation of linear operation by the matrixexpression of Equation (6), since pixel thinning is executed, pixel dataRc₆ and Rc₈ at different positions after distortion correction arerepresented by the weighted linear sums of the original pixel data Ri₆and Ri₁₀, which are given byRc ₆= 11/14Ri₆+ 3/14Ri ₁₀ Rc₈= 3/14Ri ₆+ 11/14Ri ₁₀.  (7)The pixel data used to obtain the pixel data Rc₈ is the same as thepixel data used to obtain the pixel data Rc₆. More specifically, theorder of the pixel data used to obtain the pixel data Rc₈ is differentfrom that of the pixel data used to obtain the pixel data Rc₁ to Rc₆before that (i.e., the phase is shifted). This also applies to the pixeldata Gc₇ and Gc₉.

As shown in FIG. 5, pixel data that is actually read out comprises 14data of Ri₁₀, Gi₁, Ri₂, Gi₃, Ri₄, Gi₅, Ri₆, Gi₇, Ri₁₀, Gi₁₁, Ri₁₂, Gi₁₃,Ri₁₄, and Gi₁₅. Assume that these are Rj₀, Gj₁, Rj₂, Gj₃, Rj₄, Gj₅, Rj₆,Gj₇, Rj₈, Gj₉, Rj₁₀, Gj₁₁, Rj₁₂, and Gj₁₃, respectively. That is,$\begin{matrix}{\begin{pmatrix}{Ri}_{0} \\{Gi}_{1} \\{Ri}_{2} \\{Gi}_{3} \\{Ri}_{4} \\{Gi}_{5} \\{Ri}_{6} \\{Gi}_{7} \\{Ri}_{10} \\{Gi}_{11} \\{Ri}_{12} \\{Gi}_{13} \\{Ri}_{14} \\{Gi}_{15}\end{pmatrix} = {\begin{pmatrix}{Rj}_{0} \\{Gj}_{1} \\{Rj}_{2} \\{Gj}_{3} \\{Rj}_{4} \\{Gj}_{5} \\{Rj}_{6} \\{Gj}_{7} \\{Rj}_{8} \\{Gj}_{9} \\{Rj}_{10} \\{Gj}_{11} \\{Rj}_{12} \\{Gj}_{13}\end{pmatrix}.}} & (8)\end{matrix}$

As described above, Ri_(2p) and Gi_(2p+1) (p is an integer that is nosmaller than 0 and smaller than 7) represent pixel data of pixelsarrayed in the horizontal direction in the photoelectric conversionelement 122. Inconsecutive subscripts represent data that are thinnedout in the readout. Rj_(2r) and Gj_(2r+1) (r is an integer that is nosmaller than 0 and smaller than 6) represent pixel data that areactually readout by the pixel thinning readout and have consecutivesubscripts corresponding to the readout order.

When Equation (8) is substituted into Equation (6), we obtain$\begin{matrix}{\begin{pmatrix}{Rc}_{0} \\{Gc}_{1} \\{Rc}_{2} \\{Gc}_{3} \\{Rc}_{4} \\{Gc}_{5} \\{Rc}_{6} \\{Gc}_{7} \\{Rc}_{8} \\{Gc}_{9} \\{Rc}_{10} \\{Gc}_{11} \\{Rc}_{12} \\{Gc}_{13}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & \frac{13}{14} & 0 & \frac{1}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & \frac{12}{14} & 0 & \frac{2}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & \frac{11}{14} & 0 & \frac{3}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & \frac{10}{14} & 0 & \frac{4}{14} & 0 & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & \frac{9}{14} & 0 & \frac{5}{14} & 0 & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & \frac{11}{14} & 0 & \frac{3}{14} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{21}{28} & 0 & \frac{7}{28} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & \frac{3}{14} & 0 & \frac{11}{14} & 0 & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{5}{28} & 0 & \frac{23}{28} & 0 & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{4}{14} & 0 & \frac{10}{14} & 0 & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{3}{14} & 0 & \frac{11}{14} & 0 & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{2}{14} & 0 & \frac{12}{14} & 0 \\0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & 0 & \frac{1}{14} & 0 & \frac{13}{14}\end{pmatrix}{\begin{pmatrix}{Rj}_{0} \\{Gj}_{1} \\{Rj}_{2} \\{Gj}_{3} \\{Rj}_{4} \\{Gj}_{5} \\{Rj}_{6} \\{Gj}_{7} \\{Rj}_{8} \\{Gj}_{9} \\{Rj}_{10} \\{Gj}_{11} \\{Rj}_{12} \\{Gj}_{13}\end{pmatrix}.}}} & (9)\end{matrix}$

This is distortion correction conversion that obtains 14 pixel dataafter 14/16 reduction conversion from the 14 pixel data that is actuallysequentially read out by the pixel thinning readout.

The above-described linear distortion correction can be expanded to atwo-dimensional array in the following way.

Equation (9) is expressed byC=A B.  (10)where A is the conversion matrix that executes linear distortioncompensation (i.e., in the above-described example, distortioncompensation for the 14 pixel data obtained by thinning two of 16pixels), B is a matrix with n rows and 1 column representing pixel databefore distortion compensation, and C is a matrix with n rows and 1column representing pixel data after distortion compensation. Thereadout data shown in FIG. 4 is expressed by a matrix D_(i) given by$\begin{matrix}{D_{i} = {\begin{pmatrix}\begin{matrix}{Ri}_{0,0} & {Gi}_{0,1} & {Ri}_{0,2} & {Gi}_{0,3} & {Ri}_{0,4} & {Gi}_{0,5} & {Ri}_{0,6} & {Gi}_{0,7} \\{Gi}_{1,0} & {Bi}_{1,1} & {Gi}_{1,2} & {Bi}_{1,3} & {Gi}_{1,4} & {Bi}_{1,5} & {Gi}_{1,6} & {Bi}_{1,7} \\{Ri}_{2,0} & {Gi}_{2,1} & {Ri}_{2,2} & {Gi}_{2,3} & {Ri}_{2,4} & {Gi}_{2,5} & {Ri}_{2,6} & {Gi}_{2,7} \\{Gi}_{3,0} & {Bi}_{3,1} & {Gi}_{3,2} & {Bi}_{3,3} & {Gi}_{3,4} & {Bi}_{3,5} & {Gi}_{3,6} & {Bi}_{3,7} \\{Ri}_{4,0} & {Gi}_{4,1} & {Ri}_{4,2} & {Gi}_{4,3} & {Ri}_{4,4} & {Gi}_{4,5} & {Ri}_{4,6} & {Gi}_{4,7} \\{Gi}_{5,0} & {Bi}_{5,1} & {Gi}_{5,2} & {Bi}_{5,3} & {Gi}_{5,4} & {Bi}_{5,5} & {Gi}_{5,6} & {Bi}_{5,7} \\{Ri}_{6,0} & {Gi}_{6,1} & {Ri}_{6,2} & {Gi}_{6,3} & {Ri}_{6,4} & {Gi}_{6,5} & {Ri}_{6,6} & {Gi}_{6,7} \\{Gi}_{7,0} & {Bi}_{7,1} & {Gi}_{7,2} & {Bi}_{7,3} & {Gi}_{7,4} & {Bi}_{7,5} & {Gi}_{7,6} & {Bi}_{7,7}\end{matrix} & \begin{matrix}{Ri}_{0,10} & {Gi}_{0,11} & {Ri}_{0,12} & {Gi}_{0,13} & {Ri}_{0,14} & {Gi}_{0,15} \\{Gi}_{1,10} & {Bi}_{1,11} & {Gi}_{1,12} & {Bi}_{1,13} & {Gi}_{1,14} & {Bi}_{1,15} \\{Ri}_{2,10} & {Gi}_{2,11} & {Ri}_{2,12} & {Gi}_{2,13} & {Ri}_{2,14} & {Gi}_{2,15} \\{Gi}_{3,10} & {Bi}_{3,11} & {Gi}_{3,12} & {Bi}_{3,13} & {Gi}_{3,14} & {Bi}_{3,15} \\{Ri}_{4,10} & {Gi}_{4,11} & {Ri}_{4,12} & {Gi}_{4,13} & {Ri}_{4,14} & {Gi}_{4,15} \\{Gi}_{5,10} & {Bi}_{5,11} & {Gi}_{5,12} & {Bi}_{5,13} & {Gi}_{5,14} & {Bi}_{5,15} \\{Ri}_{6,10} & {Gi}_{6,11} & {Ri}_{6,12} & {Gi}_{6,13} & {Ri}_{6,14} & {Gi}_{6,15} \\{Gi}_{7,10} & {Bi}_{7,11} & {Gi}_{7,12} & {Bi}_{7,13} & {Gi}_{7,14} & {Bi}_{7,15}\end{matrix} \\\begin{matrix}{Ri}_{10,0} & {Gi}_{10,1} & {Ri}_{10,2} & {Gi}_{10,3} & {Ri}_{10,4} & {Gi}_{10,5} & {Ri}_{10,6} & {Gi}_{10,7} \\{Gi}_{11,0} & {Bi}_{11,1} & {Gi}_{11,2} & {Bi}_{11,3} & {Gi}_{11,4} & {Bi}_{11,5} & {Gi}_{11,6} & {Bi}_{11,7} \\{Ri}_{12,0} & {Gi}_{12,1} & {Ri}_{12,2} & {Gi}_{12,3} & {Ri}_{12,4} & {Gi}_{12,5} & {Ri}_{12,6} & {Gi}_{12,7} \\{Gi}_{13,0} & {Bi}_{13,1} & {Gi}_{13,2} & {Bi}_{13,3} & {Gi}_{13,4} & {Bi}_{13,5} & {Gi}_{13,6} & {Bi}_{13,7} \\{Ri}_{14,0} & {Gi}_{14,1} & {Ri}_{14,2} & {Gi}_{14,3} & {Ri}_{14,4} & {Gi}_{14,5} & {Ri}_{14,6} & {Gi}_{14,7} \\{Gi}_{15,0} & {Bi}_{15,1} & {Gi}_{15,3} & {Bi}_{15,3} & {Gi}_{15,4} & {Bi}_{15,5} & {Gi}_{15,6} & {Bi}_{15,7}\end{matrix} & \begin{matrix}{Ri}_{10,10} & {Gi}_{10,11} & {Ri}_{10,12} & {Gi}_{10,13} & {Ri}_{10,14} & {Gi}_{10,15} \\{Gi}_{11,10} & {Bi}_{11,11} & {Gi}_{11,12} & {Bi}_{11,13} & {Gi}_{11,14} & {Bi}_{11,15} \\{Ri}_{12,10} & {Gi}_{12,11} & {Ri}_{12,12} & {Gi}_{12,13} & {Ri}_{12,14} & {Gi}_{12,15} \\{Gi}_{13,10} & {Bi}_{13,11} & {Gi}_{13,12} & {Bi}_{13,13} & {Gi}_{13,14} & {Bi}_{13,15} \\{Ri}_{14,10} & {Gi}_{14,11} & {Ri}_{14,12} & {Gi}_{14,13} & {Ri}_{14,14} & {Gi}_{14,15} \\{Gi}_{15,10} & {Bi}_{15,11} & {Gi}_{15,12} & {Bi}_{15,13} & {Gi}_{15,14} & {Bi}_{15,15}\end{matrix}\end{pmatrix}.}} & (11)\end{matrix}$

In Equation (11), the omitted portions are drawn with lines. Let D₀ be auniform 14 (pixels)×14 (pixels) array. Conversion for distortioncorrection in the vertical direction after distortion correction in thehorizontal direction is given by using A in Equation (10) byD^(C)=A A^(T)D_(i).  (12)where A^(T) is the transpose of A.

The linear distortion conversion, i.e., rewrite from Equation (4) toEquation (6) can also be considered as follows.

-   -   (1) When pixel data at a position X is read out and pixel data        at a position X+2 is also readout, the coefficients in        Equation (4) are directly used as the weighting coefficients of        pixel data.    -   (2) When the pixel data at the position X is read out and the        pixel data at the position X+2 is not readout, pixel data at a        position X+4 is read out instead. A weighting coefficient x of        the pixel data at the position X is changed to x′=0.5(x+1). The        weighting coefficient of the pixel data at the position X+4 is        the residual of the changed coefficient x′ to 1, i.e., 1−x′.    -   (3) When the pixel data at the position X is not readout and the        pixel data at the position X+2 is read out, the readout position        of the position X is shifted ahead by two to X−2. The weighting        coefficient x of the pixel data at the position X−2 is changed        to x′=0.5×. The weighting coefficient of the pixel data at the        position X+2 is the residual of the changed coefficient to 1,        i.e., 1−x′.

Hence, instead of executing distortion correction by making the pixelreadout positions correspond to the correction coefficients by using thelookup table (LUT), the distortion correction coefficients may directlybe calculated from the readout rule by using the arithmetic processingfunction of the CPU.

Distortion correction after thinning readout in the color image sensingelement with a primary color Bayer matrix has been described above. Thedistortion correction after thinning readout may be executed even for amonochrome image sensing element or another color filter matrix in asimilar manner.

When the image signal of the photoelectric conversion readout unit isdirectly stored in the memory, and the operation is performed by addressdesignation, the above-described problem of phase can be avoided.Pipeline processing at a higher speed will be described.

FIG. 6 shows an example in which two of eight pixels are thinned out inboth the horizontal and vertical directions. For example, the readout ofthe first row with thinning in the horizontal direction will beexamined. When the upper left corner in FIG. 6 is defined as areference, the readout pixel positions are Ri₀, Gi₁, Ri₂, Gi₃, Ri₄, Gi₅,Ri₆, and Gi₇. This rule is repeated. The matrix expression of distortioncorrection (conversion) in this example is given by $\begin{matrix}{\begin{pmatrix}{Rc}_{0} \\{Gc}_{1} \\{Rc}_{2} \\{Gc}_{3} \\{Rc}_{4} \\{Gc}_{5}\end{pmatrix} = {\begin{pmatrix}1 & 0 & 0 & 0 & 0 & 0 \\0 & \frac{5}{6} & 0 & \frac{1}{6} & 0 & 0 \\0 & 0 & \frac{5}{6} & 0 & \frac{1}{6} & 0 \\0 & 0 & 0 & \frac{3}{4} & 0 & \frac{1}{4} \\0 & 0 & \frac{1}{6} & 0 & \frac{5}{6} & 0 \\0 & 0 & 0 & \frac{1}{12} & 0 & \frac{11}{12}\end{pmatrix}{\begin{pmatrix}{Ri}_{0} \\{Gi}_{1} \\{Ri}_{2} \\{Gi}_{3} \\{Ri}_{6} \\{Gi}_{7}\end{pmatrix}.}}} & (13)\end{matrix}$

The pipeline processing is executed by a filter processing unit shown inFIG. 7. A shift register 362 shifts held image data by one to the rightfor each operation corresponding to the clock. A selector 364 selectsone of the first and third pixel data of five adjacent pixel data heldby the shift register 362 in accordance with the state of s1. A selector366 selects one of the third and fifth pixel data of five adjacent pixeldata held by the shift register 362 in accordance with the state of s2.

A multiplier 374 multiplies an output d1 from the selector 364 by acoefficient k1 of weighted addition. A multiplier 376 multiplies anoutput d2 from the selector 366 by a coefficient k2 of weightedaddition. An adder 378 adds the output from a multiplier 394 and theoutput from a multiplier 396.

Table 1 shows the operation (state transition) of pipeline processing bythe filter processing unit shown in FIG. 7. TABLE 1 c1 c2 c3 c4 c5 c6 s1s2 d1 d2 k1 k2 out i0 i1 i2 i3 i4 i5 0 0 i2 i4 1 0 1 × i2 + 0 × i4 i1 i2i3 i4 i5 i6 0 0 i3 i5 ⅚ ⅙ ⅚ × i3 + ⅙ × i5 i2 i3 i4 i5 i6 i7 0 0 i4 i6 ⅚⅙ ⅚ × i4 + ⅙ × i6 i3 i4 i5 i6 i7 i8 0 0 i5 i7 ¾ ¼ ¾ × i5 + ¼ × i7 i4 i5i6 i7 i8 i9 1 1 i4 i6 ⅙ ⅚ ⅙ × i4 + ⅚ × i6 i5 i6 i7 i8 i9 i10 1 1 i5 i71/12 11/12 1/12 × i5 + 11/12 × i7 i6 i7 i8 i9 i10 i11 0 0 i8 i10 1 0 1 ×i8 + 0 × i10 i7 i8 i9 i10 i11 i12 0 0 i9 i11 ⅚ ⅙ ⅚ × i9 + ⅙ × i11 i8 i9i10 i11 i12 i13 0 0 i10 i12 ⅚ ⅙ ⅚ × i10 + ⅙ × i12 i9 i10 i11 i12 i13 i140 0 i11 i13 ¾ ¼ ¾ × i11 + ¼ × i13 i10 i11 i12 i13 i14 i15 1 1 i10 i12 ⅙⅚ ⅙ × i10 + ⅚ × i12 i11 i12 i13 i14 i15 i16 1 1 i11 i13   1/12 11/121/12 × i11 + 11/12 × i13 i12 i13 i14 i15 i16 i17 0 0 i14 i16 1 0 1 ×i14 + 0 × i16 i13 i14 i15 i16 i17 i18 0 0 i15 i17 ⅚ ⅙ ⅚ × i15 + ⅙ × i17i14 i15 i16 i17 i18 i19 0 0 i16 i18 ⅚ ⅙ ⅚ × i16 + ⅙ × i18 i15 i16 i17i18 i19 i20 0 0 i17 i19 ¾ ¼ ¾ × i17 + ¼ × i19 i16 i17 i18 i19 i20 i21 11 i16 i18 ⅙ ⅚ ⅙ × i16 + ⅚ × i18 i17 i18 i19 i20 i21 i22 1 1 i17 i19  1/12 11/12 1/12 × i17 + 11/12 × i19 i18 i19 i20 i21 i22 i23 0 0 i20 i221 0 1 × i20 + 0 × i22 i19 i20 i21 i22 i23 i24 0 0 i21 i23 ⅚ ⅙ ⅚ × i21 +⅙ × i23 i20 i21 i22 i23 i24 i25 0 0 i22 i24 ⅚ ⅙ ⅚ × i22 + ⅙ × i24

The pixel data sequence (i0, i1, i2, . . . ) supplied to the shiftregister 362 shifts to the right for each operation according to theclock from the initial state in which c1=i0, c2=i1, and c3=i2.Accordingly, the selector 364 selects c3 when s1 is 0 (d1=c3) and c1when s1 is 1 (d1=c1). On the other hand, the selector 366 selects c5when s2 is 0 (d2=c5) and c3 when s2 is 1 (d2=c3). In synchronism withthe clock, the coefficient k1 is supplied from the memory in the filtercoefficient setting unit 154 to the multiplier 374, and the coefficientk2 is supplied to the multiplier 376. The adder 378 outputsout=k1×d1+k2×d2.

As is apparent from Table 1, when sequential data shift, selectorswitching according to the states of s1 and s2, outputs of the weightingcoefficients k1 and k2 corresponding to the thinning rule indicated byEquation (6), and weighted addition operation are performedsynchronously, pipeline processing including pixel phase operation(selector switching) is executed.

When there is only one kind of color information on a single imagesensing element, as in a monochrome or 3-chip color image sensingelement unlike an image sensing element having color filter arrays (CFA)such as a Bayer matrix or a complementary color filter matrix, theinterval of one pixel need not be prepared to process pixels of the samecolor in distortion correction, unlike the CFA of a Bayer matrix. Inaddition, thinning need not be executed for two consecutive pixels tomake signals of the same color have the same phase (to execute thereadout in a predetermined order of, e.g., R, G, R, G . . . .)

In a single-color image sensing element, the weighed addition ofadjacent pixels is done while selecting data. Hence, distortioncorrection (conversion) is expressed byPc ₂ =aPi ₁+(1−a)Pi ₂,  (14)where Pc is pixel data after conversion, and Pi is pixel data as theconversion source.

The pipeline processing is executed by a filter processing unit shown inFIG. 8. A shift register 382 shifts held image data by one to the rightfor each operation corresponding to the clock. A selector 368 selectsone of the first and second pixel data (C1 and C2) of three adjacentpixel data in accordance with the state of s1. A selector 386 selectsone of the second and third pixel data (C2 and C3) of three adjacentpixel data in accordance with the state of s2.

A multiplier 394 multiplies the output d1 from the selector 384 by thecoefficient k1 of weighted addition. A multiplier 396 multiplies theoutput d2 from the selector 386 by the coefficient k2 of weightedaddition. An adder 378 adds the output from the multiplier 394 and theoutput from the multiplier 396.

The image sensing apparatus of this embodiment, in the thinning readoutmode, reads out image data acquired by the photoelectric conversionelement thinning out it in the hardware in at least vertical directionand, preferably, in both the horizontal and vertical directions. Hence,the image sensing apparatus of this embodiment can read out image datain a shorter time than a normal image sensing apparatus that reads outall image data from the photoelectric conversion elements and then thinsout the image data by using software.

The suitable adaptive range of size change by thinningreadout+distortion correction processing and the advantages of switchingthe readout scheme (full-pixel readout mode, averaging readout mode, andthinning readout mode) and subsequent image signal processing inaccordance with the image scaling ratio will be described below.

Tables 2 to 4 show sensory evaluation of typical objects so as toindicate the superiority in image quality between (1) images by thinningreadout+distortion correction processing and (2) images by averagingreadout+linear enlargement processing at various reductionmagnifications. More specifically, Table 2 shows the evaluation resultof objects such as a building and a structure. Table 3 shows theevaluation result of objects such as a person and a natural landscape.Table 4 shows the evaluation result of objects such as a resolutionchart and a line drawing. In these tables, “1” represents that thinningreadout+distortion correction processing is superior, and “2” representsthat averaging readout+linear enlargement processing is superior. Inaddition, “A” represents that both processing operations are equal.TABLE 2 Object: Building, Structure Reduction Luminance Colormagnification Contrast Resolution Distortion moiré moiré 93% 1 1 1 1 191% 1 1 1 1 1 87% 1 1 1 1 1 83% 1 1 1 1 1 80% 1 1 1 1 1 75% 1 1 Δ 1 171% Δ Δ 2 Δ Δ 66% 2 2 2 2 2 60% 2 2 2 2 2

TABLE 3 Object: Person, Natural landscape Reduction Luminance Colormagnification Contrast Resolution Distortion moiré moiré 93% 1 1 1 1 191% 1 1 1 1 1 87% 1 1 1 1 1 83% 1 1 1 1 1 80% 1 1 1 1 1 75% 1 1 1 1 171% 1 Δ Δ Δ Δ 66% Δ 2 2 2 2 60% 2 2 2 2 2

TABLE 4 Object: Resolution chart, Line drawing Reduction Luminance Colormagnification Contrast Resolution Distortion moiré moiré 93% 1 1 1 1 191% 1 1 1 1 1 87% 1 1 1 1 1 83% 1 1 1 1 1 80% 1 1 1 1 1 75% 1 1 Δ 1 171% Δ Δ 2 Δ Δ 66% 2 2 2 2 2 60% 2 2 2 2 2

Reasoning of the sensory evaluation will briefly be described here.

When the reduction magnification is high, e.g., in the mode to read out14 of 16 pixels (corresponding to conversion given by Equation (6)), thequality of the images obtained by thinning readout+distortion correctionprocessing bears comparison with that of the images obtained byprocessing of changing the size by linear interpolation after thefull-pixel readout (corresponding to conversion given by Equation (1)).However, when the reduction magnification is low, e.g., in the mode toread out four of six pixels, the quality is poor. A possible reason forthis is as follows. Since the ratio of pixels thinned out increases asthe reduction magnification decreases, the ratio of omitted imageinformation increases. Hence, reconstruction by linear interpolationrepresented from Equation (6) is difficult.

The images obtained by averaging readout+linear enlargement processingare inferior in resolution and contrast. However, when the reductionmagnification is low (e.g., 60%), the degradation in band by enlargement(in ½ sampling, original data is reduced to 50%; to obtain a 60% image,120% enlargement is executed) is small.

In consideration of the above facts, when the reduction magnification ishigh, the thinning readout mode is preferably selected. When thereduction magnification is low, averaging readout+enlargement by linearinterpolation is preferably selected.

The type of object will be examined. For the object of a natural image,importance is placed on the texture and resolution. For the object of anatural image, thinning readout+distortion correction processing issuperior to averaging readout+linear enlargement processing from theviewpoint of band degradation and moiré reduction. Hence, for the objectof a natural image, the reduction magnification to switch between thethinning readout mode and the averaging readout mode is preferably setrelatively low (e.g., approximately 66%).

For an object such as a line drawing, importance is placed on reductionof distortion because the contrast and resolution can be recovered tosome extent by another image processing (band enhancement using afilter). For this reason, for an object such as a line drawing,averaging readout+linear enlargement processing is superior to thinningreadout+distortion correction processing. Hence, for an object such as aline drawing, the reduction magnification to switch between the thinningreadout mode and the averaging readout mode is preferably set relativelyhigh (e.g., approximately 75%).

By switching between the thinning readout mode and averaging readoutmode (and subsequent image signal processing) in accordance with thereduction magnification, a high-resolution image can be obtained in awide reduction magnification range.

As another advantage of switching between the readout scheme andsubsequent image signal processing in accordance with the reductionmagnification, the number of magnification steps can be increased in awide magnification range.

A scaling ratio X in the thinning readout is given by $\begin{matrix}{X = \frac{n - k}{n}} & (15)\end{matrix}$(n: unit of thinning block, k: number of pixels to be thinned out). Thatis, the scaling ratio X is given by a ratio of integers.

When the reduction magnification is relatively high, size change can bedone at a relatively fine interval from, e.g., 20/22=91% (mode to readout 20 of 22 pixels) to 18/20=90%, 16/18=89%, 14/16=87.5%, 12/14=85.7%,and 10/12=83.3%.

However, when the reduction magnification is relatively low, the scalingratio interval becomes large, like, e.g., 6/8=75%, 10/14=71%, 4/6=66%,and 6/10=60%. Although a reduction magnification such as 8/14=57% isalso possible, reconstruction by distortion correction by linearinterpolation becomes difficult because the ratio of pixels to bethinned out in a block increases.

Hence, by switching between the thinning readout mode and averagingreadout mode (and subsequent image signal processing) in accordance withthe reduction magnification, the reduction magnification can bedesignated at a fine interval.

FIG. 9 shows size change corresponding to thinning readout+distortioncorrection processing. More specifically, FIG. 9 shows an example inwhich a 720×540 pixel area is changed to VGA (640×480) on thephotoelectric conversion element. In this example, the rule of readingout 16 pixels by thinning out two of 18 pixels is repeated. The imagesize is reduced to 16/18= 8/9, i.e., about 89% (720→640, and 540→480).

FIG. 10 shows size change corresponding to averaging readout+linearinterpolation processing. More specifically, FIG. 10 shows an example inwhich 1120×840 pixels is changed to the VGA size. In this example,horizontal 2-pixel averaging readout+vertical 2-line interlaced readoutis executed in accordance with the VGA clock, thereby reading out anarea of 1280×960 pixels in advance. A portion corresponding to an areaof 1120×840 pixels is extracted and changed to the VGA size. Since2-pixel averaging is executed, the memory area corresponding to the areaof 1120×840 pixels on the photoelectric conversion element is 560×420.This area is extracted and enlarged by linear interpolation to changethe size ( 640/560 =about 1.14 times). At this time, the image size isreduced to 640/1120=57% on the basis of the image size on thephotoelectric conversion element.

FIG. 11 shows readout scheme switching and size change corresponding tothe scaling ratio in the image sensing apparatus of this embodiment. Fora scaling ratio of 100% or more, size change is executed by full-pixelreadout+enlargement processing by linear interpolation.

For a scaling ratio or reduction ratio of approximately 70% to 100%,size change is executed by thinning readout+distortion correctionprocessing. For a scaling ratio or reduction ratio lower than 70%, sizechange is executed by 2:1 averaging readout+linear interpolationprocessing. The readout area on the image sensing element in the 2:1averaging readout is twice the readout area on the image sensing elementin the full-pixel readout.

As is apparent from the above description, in the image sensingapparatus according to this embodiment, a high-resolution image can beobtained in a wide scaling ratio range by switching between the readoutscheme and subsequent image signal processing in accordance with theimage scaling ratio. In addition, at a scaling ratio lower than 100%,the scaling ratio can be designated at a fine interval.

Second Embodiment

This embodiment is particularly directed to an image sensing apparatusthat is suitably used to sense a moving image.

FIG. 12 shows the arrangement of an image sensing apparatus according tothe second embodiment of the present invention. The same referencenumerals as in the image sensing apparatus 100 of the first embodimentdenote the same elements in FIG. 12, and a detailed description thereofwill be omitted to avoid any repetitive description.

An image sensing apparatus 200 of this embodiment has an imaging opticalsystem 110 that forms an optical image of an object and an image sensingdevice 220 that continuously outputs an image signal in a predeterminedregion of the optical image formed by the imaging optical system 110.That is, the image signal output from the image sensing device 220 is amoving image signal, which comprises image data of a plurality of framesthat time-serially continue.

The image sensing device 220 has a two-dimensional photoelectricconversion element 222 that photoelectrically converts the optical imageformed by the imaging optical system 110 to acquire image data (pixeldata set) and a readout control unit 224 that continuously reads out theimage data acquired by the photoelectric conversion element 222 inaccordance with a supplied readout rule.

The image sensing apparatus 200 also has an image scaling ratioselection unit 132 that selects the scaling ratio of the image to beoutput, a readout scheme selection unit 234 that selects, in accordancewith the selected image scaling ratio, the readout scheme of the imagedata to be read out from the photoelectric conversion element 222 by thereadout control unit 224, and a readout rule supply unit 240 thatsupplies, to the image sensing device 220, a readout rule correspondingto the readout scheme selected by the readout scheme selection unit 234.

The readout scheme selection unit 234 selects one of the thinningreadout mode, horizontal averaging vertical interlaced readout mode, andvertical interlaced readout mode in accordance with the selected imagescaling ratio. The readout rule supply unit 240 includes a thinningreadout rule setting unit 242 that sets a readout rule corresponding tothe thinning readout mode, a horizontal averaging vertical interlacedreadout rule setting unit 244 that sets a readout rule corresponding tothe horizontal averaging vertical interlaced readout mode, and avertical interlaced readout rule setting unit 246 that sets a readoutrule corresponding to the vertical interlaced readout mode.

More specifically, for an image scaling ratio lower than 100%, i.e.,image reduction, the readout scheme selection unit 234 selects one ofthe thinning readout mode and horizontal averaging vertical interlacedreadout mode. For an image scaling ratio of 100% or more, i.e., imageenlargement, the readout scheme selection unit 234 selects the verticalinterlaced readout mode.

For an image scaling ratio lower than 100%, the readout scheme selectionunit 234 selects one of the thinning readout mode and horizontalaveraging vertical interlaced readout mode on the basis of the imagescaling ratio and an important one of image quality factors such ascontrast, resolution, distortion, luminance moiré, and color moiré ofthe image to be output.

The image sensing apparatus 200 also has a distortion correction unit150 that executes distortion correction for the image signal output fromthe image sensing device in the thinning readout mode, a readout phasecontrol unit 252 that changes, for each frame, the reference position ofan area (readout area) of image data to be read out from thephotoelectric conversion element 222 by the readout control unit 224 inthe thinning readout mode, and an image area selection processing unit254 that selects an area of a region common to all frames of thecorrected image signal output from the distortion correction unit 150 onthe basis of the reference position of the readout area that is changedfor each frame by the readout phase control unit 252 in the thinningreadout mode. Details of the distortion correction unit 150 are the sameas described in the first embodiment.

The readout phase control unit 252 changes the reference position of thereadout area for each frame, and the image area selection processingunit 254 selects the area of a region common to all frames. Accordingly,the thinning readout rule setting unit 242 sets a readout rule so thatthe readout control unit 224 reads out image data in a wider area thanthe region of the image to be output.

The readout control unit 224 in the image sensing device 220continuously reads out image data (pixel data of one frame) in acorresponding area in the pixel matrix in the photoelectric conversionelement 222 on the basis of the readout rule set by the thinning readoutrule setting unit 242 and the reference position of the readout area setby the readout phase control unit 252. As a result, the image sensingdevice 220 outputs a moving image signal comprising image data of aplurality of frames that time-serially continue.

The image sensing apparatus 200 also has a plurality of storage units,e.g., two frame memories 272 and 274, that temporarily store image dataof a plurality of frames of the image signal output from the image areaselection processing unit 254 in the thinning readout mode or the imagesignal output from the image sensing device 220 in the horizontalaveraging vertical interlaced readout mode and vertical interlacedreadout mode.

The image sensing apparatus 200 also has an interframe arithmeticprocessing unit 282 that generates new image data by executing weightedaveraging for the image data of a plurality of frames stored in theframe memories 272 and 274 in the thinning readout mode.

The image sensing apparatus 200 also has an interframe interpolationarithmetic processing unit 284 that executes interframe interpolationfor image data of a plurality of frames stored in the frame memories 272and 274 in the horizontal averaging vertical interlaced readout mode andvertical interlaced readout mode (in the interlaced readout, positionand line data are omitted between continuous frames; the data (linedata) of the omitted portions are compensated by adjacent frame data)and a linear interpolation size change unit 286 that executes sizechange by linear interpolation in accordance with the image scalingratio for the image signal output from the interframe interpolationarithmetic processing unit 284 in the horizontal averaging verticalinterlaced readout mode and vertical interlaced readout mode.

In addition, the image sensing apparatus 200 has a first selector 262that selectively sends the image signal output from the image sensingdevice 220 to one of the distortion correction unit 150 and framememories 272 and 274 in accordance with the readout scheme selected bythe readout scheme selection unit 234 and second selectors 276 and 278that selectively send the image signals respectively from the framememories 272 and 274 to one of the interframe arithmetic processing unit282 and interframe interpolation arithmetic processing unit 284.

When the readout scheme selection unit 234 selects the thinning readoutmode, the first selector 262 sends the image signal from the imagesensing device 220 to the frame memories 272 and 274 through thedistortion correction unit 150 and image area selection processing unit254. The second selectors 276 and 278 send the image signalsrespectively from the frame memories 272 and 274 to the image signalprocessing unit 172 through the interframe arithmetic processing unit282.

On the other hand, when the readout scheme selection unit 234 selectsthe horizontal averaging vertical interlaced readout mode or verticalinterlaced readout mode, the first selector 262 sends the image signalfrom the image sensing device 220 directly to the frame memories 272 and274. The second selectors 276 and 278 send the image signalsrespectively from the frame memories 272 and 274 to the image signalprocessing unit 172 through the interframe interpolation arithmeticprocessing unit 284 and linear interpolation size change unit 286.

In moving image sensing by a conventional video system, an interlacedscanning method is often used, in which 2 fields=1 frame. Image flickerby interlaced scanning is unnoticeable at a general frame rate of 1/30.If anything, with the interfaced scanning, image information in a wideregion can be obtained within the same time as in full scanning, and ahigh-resolution image can be obtained at a high speed by interpolationbetween fields.

In the image sensing apparatus 200 of this embodiment, in the verticalinterlaced readout mode, pixel data is alternately read out in everyother line unit for every two fields that are adjacent time-serially toform one frame in accordance with the generally well-known interlacedscanning method. As a result, an image with few flicker can be obtainedby the effect of interpolation between line units.

The line unit indicates the base unit of repetition along thearrangement of lines in the filter array. In, e.g., the Bayer matrix,the line unit includes two actual pixel lines.

In the image sensing apparatus 200 of this embodiment, in the horizontalaveraging vertical interlaced readout mode, pixel data is read out inevery other line unit for every two fields that are adjacenttime-serially to form one frame while averaging two corresponding pixeldata for every two pixel units in each line.

The pixel unit indicates the base unit of repetition along each line inthe filter array. In, e.g., the Bayer matrix, one pixel unit includestwo pixels (in, e.g., FIG. 13, a set of R and G of the first line).

FIG. 13 schematically shows pixel data of two fields that are adjacenttime-serially to form one frame, the pixel data being read out by thehorizontal averaging vertical interlaced readout.

As shown in FIG. 13, in each field, pixel data is read out in everyother line unit, i.e., every other line pair in each field. In eachline, two corresponding pixel data for every two pixel units, i.e., twopixel data of the same type for every four pixels are read out whileaveraging them by a weight of ½:½. That is, two pixel data correspondingto four pixels are read out by one clock (CLK).

As a result of such readout, an image with few flicker is obtained bythe effect of interpolation between pixel units and between line units.

In the image sensing apparatus 200 of this embodiment, in the thinningreadout mode, the interlaced scanning interpolates omitted pixel databetween two fields so that the omitted pixel data is interpolatedbetween two consecutive frames.

For this purpose, the readout phase control unit 252 changes, for eachframe, the reference position of the area (readout area) of pixel datato be read out with thinning-out from the photoelectric conversionelement 222 by the readout control unit 224. More specifically, thereadout phase control unit 252 periodically changes, for each frame, thereference position of the readout area in accordance with apredetermined rule.

As a consequence, pixel data at specific positions in the photoelectricconversion element 222 that is omitted for thinning readout in the imagedata of a specific frame is contained in the image data of anotherframe. That is, it can be avoided that pixel data at specific positionsin the photoelectric conversion element 222 is always omitted from theimage signal output from the image sensing device 220.

FIG. 14 schematically shows pixel data in the same area of two frames(frame A and frame B) that are adjacent time-serially, the pixel databeing read out by 10/12 thinning readout and readout area referenceposition shift processing.

As is apparent from FIG. 14, for example, the image data (pixel dataset) of the frame B contains pixel data that are skipped in reading outthe image data of the frame A, i.e., pixel data omitted in the imagedata of the frame A. That is, the image data of two framescomplementarily contain each other's omitted pixel data.

The interframe arithmetic processing unit 282 executes processing ofinterpolating omitted pixel data for the image data of consecutiveframes stored in the frame memories 272 and 274. For example, additionof ½:½ is executed for the image data of two consecutive frames. As aresult, the same effect as in interframe interpolation, i.e.,deinterlacing processing by the well-known interlaced scanning method isobtained, and an image with few flicker is obtained.

FIGS. 15 and 16 schematically show the shift of the reference positionof the readout area in a readout by repeating 6/8 thinning readout.Referring to FIGS. 15 and 16, [x, y] represents the pixel position ofthe pixel matrix in the photoelectric conversion element 122, and (x, y)represents the pixel data array in the readout area.

As shown in FIGS. 15 and 16, the number of pixels of the photoelectricconversion element 122 is k in the horizontal direction and 1 in thevertical direction. Hence, the position of the pixel at the upper leftof the photoelectric conversion element 122 is expressed as [0, 0], andthe position of the pixel at the lower right is expressed as [k, 1]. Thenumber of pixels in the readout area of one frame is m in the horizontaldirection and n in the vertical direction. Hence, the readout startposition at the upper left of the frame is expressed as (0, 0), and thereadout end position at the lower right is expressed as (m, n). Theframe readout area shown in FIG. 16 is shifted by +2 pixels in thehorizontal direction and +2 pixels in the vertical direction from theframe readout area shown in FIG. 15.

In the frame shown in FIG. 15, the readout start position (0, 0) at theupper left matches the upper left pixel position [0, 0] of thephotoelectric conversion element 122. That is,(0, 0)=[0, 0].  (16)The readout end position (m, n) is given by(m,n)=[k−2, l−2].  (17)On the other hand, in the frame shown in FIG. 16, the readout startposition at the upper left is given by(0, 0)=[2,2].  (18)The readout end position is given by(m, n)=[k,l].  (19)

The image area selection processing unit 254 selects an area common tothe frame shown in FIG. 15 and that shown in FIG. 16. That is, for theframe shown in FIG. 15, the image area selection processing unit 254selects a rectangular area having (2, 2) and (m, n) as the diagonalapices. For the frame shown in FIG. 16, the image area selectionprocessing unit 254 selects a rectangular area having (0, 0) to (m−2,n−2) as the diagonal apices. The area selected by the image areaselection processing unit 254 always has (m−2)×(n−2) pixel data.

When the region to be cropped in advance is taken into consideration,the total number of images to be read out from the photoelectricconversion element 222 must take the output image size and phase shiftamount into consideration. The image area selection processing unit 254changes the crop area on the basis of the information of the readoutstart position.

The frame memories 272 and 274 are first in first out (FIFO) memories.The interframe arithmetic processing unit 282 generates an output imageby using pixels at the same position in the frame memories 272 and 274.

For, e.g., two frames, a synthetic image out(i, j) is given by out(i,j)=0.51(k, i, j)+0.51(k−1, i, j), (20) where i, j is the pixel position,and I(k, i, j) is the intensity of the image signal at the pixelposition i, j of the kth frame.

For three frames, the synthetic image out(i, j) is given byout(i, j)=0.25l(k, i, j)+0.5l(k−1, i, j)+0.25l(k−2, i, j)  (21)by using weighted distribution. By executing interframe interpolation,an effect of increasing the image quality by low-pass operation isobtained in addition to the distortion correction effect.

In this embodiment, thinning readout is executed in both the horizontaland vertical directions, and linear distortion correction is executed bypipeline processing in both the horizontal and vertical directions. Fora CCD, an image sensing element that executes an operation of verticaltransfer horizontal transfer cannot read out image data thinning out itin the horizontal direction in principle. For this reason, in thehorizontal direction, all pixels must be read out, and a size change bylinear interpolation must be executed, as in Equation (1).

In the image sensing apparatus 200 of this embodiment, due to the samereason as in the first embodiment, for an image scaling ratio of 100% ormore, size change is executed by vertical interlaced readout+linearinterpolation processing interpolation. For a relatively high reductionmagnification, e.g., a scaling ratio of approximately 70% to 100%, sizechange is executed by thinning readout+distortion correction processing.For a relatively low reduction magnification, e.g., a scaling ratiolower than 70%, size change is executed by horizontal averaging verticalinterlaced readout+linear interpolation processing. As in the firstembodiment, the scaling ratio to switch the readout mode may be adjustedin accordance with the object.

As a result, in the image sensing apparatus of this embodiment, byswitching between the readout scheme and subsequent image signalprocessing in accordance with the image scaling ratio, a high-resolutionimage can be obtained in a wide scaling ratio range. In addition, whenthe scaling ratio is lower than 100%, the scaling ratio can bedesignated at a fine interval.

Third Embodiment

This embodiment is particularly directed to an image sensing apparatusthat is suitably used to sense a moving image.

FIG. 17 shows the arrangement of an image sensing apparatus according tothe third embodiment of the present invention. The same referencenumerals as in the image sensing apparatus 100 of the first embodimentdenote the same elements in FIG. 17, and a detailed description thereofwill be omitted to avoid any repetitive description.

An image sensing apparatus 300 of this embodiment has an imaging opticalsystem 110 that forms an optical image of an object and an image sensingdevice 220 that continuously outputs an image signal in a predeterminedregion of the optical image formed by the imaging optical system 110.That is, the image signal output from the image sensing device 220 is amoving image signal, which comprises image data of a plurality of framesthat time-serially continue.

The image sensing device 220 has a two-dimensional photoelectricconversion element 222 that photoelectrically converts the optical imageformed by the imaging optical system 110 to acquire image data (pixeldata set) and a readout control unit 224 that continuously reads out theimage data acquired by the photoelectric conversion element 222 inaccordance with a supplied readout rule.

The image sensing apparatus 300 also has an image scaling ratioselection unit 132 that selects the scaling ratio of the image to beoutput, a readout scheme selection unit 234 that selects, in accordancewith the selected image scaling ratio, a readout scheme of the imagedata to be read out from the photoelectric conversion element 222 by thereadout control unit 224, and a readout rule supply unit 240 thatsupplies, to the image sensing device 220, a readout rule correspondingto the readout scheme selected by the readout scheme selection unit 234.

The readout scheme selection unit 234 selects one of a thinning readoutmode, a horizontal averaging vertical interlaced readout mode, and avertical interlaced readout mode in accordance with the selected imagescaling ratio. The readout rule supply unit 240 includes a thinningreadout rule setting unit 242 that sets a readout rule corresponding tothe thinning readout mode, a horizontal averaging vertical interlacedreadout rule setting unit 244 that sets a readout rule corresponding tothe horizontal averaging vertical interlaced readout mode, and avertical interlaced readout rule setting unit 246 that sets a readoutrule corresponding to the vertical interlaced readout mode.

More specifically, for an image scaling ratio lower than 100%, i.e.,image reduction, the readout scheme selection unit 234 selects one ofthe thinning readout mode and horizontal averaging vertical interlacedreadout mode. For an image scaling ratio of 100% or more, i.e., imageenlargement, the readout scheme selection unit 234 selects the verticalinterlaced readout mode.

For an image scaling ratio lower than 100%, the readout scheme selectionunit 234 selects one of the thinning readout mode and horizontalaveraging vertical interlaced readout mode on the basis of the imagescaling ratio and an important one of image quality factors such ascontrast, resolution, distortion, luminance moiré, and color moiré ofthe image to be output.

The image sensing apparatus 300 also has a timing generator 370 thatgenerates a frame timing and a readout rule change unit 372 that changesthe readout rule between frames in synchronism with the frame timingfrom the timing generator 370. When the thinning readout mode isselected, the readout rule change unit 372 changes the thinning readoutrule for each frame. Simultaneously, the readout rule change unit 372generates an instruction to select a coefficient used for a distortioncorrection filter in accordance with the readout rule. Details of adistortion correction unit 150 are the same as described in the firstembodiment.

In the readout rule that is changed for each frame by the readout rulechange unit 372, when the selected scaling ratio is constant, the numberof pixels to be read out does not change, although the referenceposition of the readout area and array pattern change.

The readout control unit 224 in the image sensing device 220continuously reads out image data (pixel data of one frame) in acorresponding area in the pixel matrix in the photoelectric conversionelement 222 on the basis of the readout rule set by the thinning readoutrule setting unit 242 and the reference position of the readout area setby the readout rule change unit 372. As a result, the image sensingdevice 220 outputs a moving image signal comprising image data of aplurality of frames that time-serially continue.

The image sensing apparatus 300 also has an interframe arithmeticprocessing unit 380 that generates new image data by using image data ofa plurality of frames. In the horizontal averaging vertical interlacedreadout mode and vertical interlaced readout mode, the interframearithmetic processing unit 380 executes interframe interpolationarithmetic processing. In the thinning readout mode, the interframearithmetic processing unit 380 executes interframe weighted additionrepresented by Equation (20) or (21) described above.

The image sensing apparatus 300 also has a selector 392 that selectivelysends the image signal output from the image sensing device 220 to oneof the distortion correction unit 150 and interframe arithmeticprocessing unit 380 in accordance with the readout scheme selected bythe readout scheme selection unit 234.

When the readout scheme selection unit 234 selects the thinning readoutmode, the selector 392 sends the image signal from the image sensingdevice 220 to the interframe arithmetic processing unit 380 through thedistortion correction unit 150.

When the readout scheme selection unit 234 selects the horizontalaveraging vertical interlaced readout mode or vertical interlacedreadout mode, the selector 392 sends the image signal from the imagesensing device 220 to the interframe arithmetic processing unit 380.

The interframe arithmetic processing unit 380 is connected to a linearinterpolation size change unit 397 through a selector 395. When thereadout scheme selection unit 234 selects the horizontal averagingvertical interlaced readout mode or vertical interlaced readout mode,and size change is necessary, the selector 395 transmits the output datafrom the interframe arithmetic processing unit 380 to the linearinterpolation size change unit 397, so that an image with apredetermined size is obtained by linear interpolation processing in theframe.

When the readout scheme selection unit 234 selects the thinning readoutmode, and the image size obtained by the distortion correction unit 150is to be used as the final size, the output from the interframearithmetic processing unit 380 is directly used. Hence, the selector 395bypasses the linear interpolation size change unit 397. To obtain animage size different from that obtained by size change by the thinningreadout, the selector 395 transmits the output data from the interframearithmetic processing unit 380 to the linear interpolation size changeunit 397, so that an image with a predetermined size is obtained bylinear interpolation processing in the frame.

As is apparent from the above description, in the image sensingapparatus according to this embodiment, a high-resolution image can beobtained in a wide scaling ratio range by switching the readout schemeand subsequent image signal processing in accordance with the imagescaling ratio. In addition, at a scaling ratio lower than 100%, thescaling ratio can be designated at a fine interval. Furthermore, in thethinning readout mode, image information in a wide region is obtainedwithin the same time as in full scanning.

The embodiments of the present invention have been described above withreference to the accompanying drawings. However, the present inventionis not limited to these embodiments, and various changes andmodifications may be made without departing from the spirit and scope ofthe present invention.

1. An image sensing apparatus that outputs an image of an object,comprising: an image sensing device including a photoelectric conversionelement that photoelectrically converts an optical image to acquireimage data, and a readout control unit that reads out, in accordancewith a supplied readout rule, the image data acquired by thephotoelectric conversion element; an image scaling ratio selection unitthat selects a scaling ratio of the image to be output; a readout schemeselection unit that selects, in accordance with the selected imagescaling ratio, a readout scheme of the image data to be read out fromthe photoelectric conversion element by the readout control unit; and areadout rule supply unit that supplies, to the readout control unit, areadout rule corresponding to the readout scheme selected by the readoutscheme selection unit.
 2. An image sensing apparatus according to claim1, wherein the readout scheme includes a thinning readout mode, anaveraging readout mode, and a full-pixel readout mode, the readoutscheme selection unit selects one readout scheme of the thinning readoutmode, averaging readout mode, and full-pixel readout mode in accordancewith the selected image scaling ratio, and the readout rule supply unitincludes a thinning readout rule setting unit that sets a readout rulecorresponding to the thinning readout mode, an averaging readout rulesetting unit that sets a readout rule corresponding to the averagingreadout mode, and a full-pixel readout rule setting unit that sets areadout rule corresponding to the full-pixel readout mode.
 3. An imagesensing apparatus according to claim 2, further comprising a distortioncorrection unit that executes distortion correction for the image signaloutput from the image sensing device in the thinning readout mode, alinear interpolation size change unit that executes size change bylinear interpolation for the image signal output from the image sensingdevice in the averaging readout mode and full-pixel readout mode, and aselector that selectively sends the image signal output from the imagesensing device to one of the distortion correction unit and the linearinterpolation size change unit in accordance with the readout schemeselected by the readout scheme selection unit.
 4. An image sensingapparatus according to claim 1, wherein the image signal output from theimage sensing device comprises a moving image signal comprising imagedata of a plurality of frames that time-serially continue, the readoutscheme selection unit selects one readout scheme of a thinning readoutmode, a horizontal averaging vertical interlaced readout mode, and ahorizontal full-pixel vertical interlaced readout mode in accordancewith the selected image scaling ratio, and the readout rule supply unitincludes a thinning readout rule setting unit that sets a readout rulecorresponding to the thinning readout mode, a horizontal averagingvertical interlaced readout rule setting unit that sets a readout rulecorresponding to the horizontal averaging vertical interlaced readoutmode, and a vertical interlaced readout rule setting unit that sets areadout rule corresponding to the vertical interlaced readout mode. 5.An image sensing apparatus according to claim 1, wherein the readoutscheme includes a thinning readout mode, and the image signal outputfrom the image sensing device contains a plurality of frames, andfurther comprising a distortion correction unit that executes distortioncorrection for the image signal output from the image sensing device inthe thinning readout mode, a readout phase control unit that changes,for each frame, a reference position of an area of the image data to beread out from the photoelectric conversion element by the readoutcontrol unit, and an image area selection processing unit that selectsan area of a region common to all frames of the corrected image signaloutput from the distortion correction unit on the basis of the referenceposition of the readout area that is changed for each frame by thereadout phase control unit.
 6. An image sensing apparatus according toclaim 5, further comprising a plurality of storage units thattemporarily store the image data of the plurality of frames of the imagesignal output from the image area selection processing unit in thethinning readout mode, and an interframe arithmetic processing unit thatgenerates new image data by executing weighted averaging for the imagedata of the plurality of frames stored in the storage units.
 7. An imagesensing apparatus according to claim 4, further comprising a pluralityof storage units that temporarily store the image data of the pluralityof frames of the image signal output from the image sensing device inthe horizontal averaging vertical interlaced readout mode and horizontalfull-pixel vertical interlaced readout mode, an interframe interpolationarithmetic processing unit that executes interframe interpolation forthe image data of the plurality of frames stored in the storage units,and a linear interpolation size change unit that executes, in accordancewith the image scaling ratio, size change by linear interpolation forthe image signal output from the interframe interpolation arithmeticprocessing unit.
 8. An image sensing apparatus according to claim 4,further comprising a distortion correction unit that executes distortioncorrection for the image signal output from the image sensing device inthe thinning readout mode, a readout phase control unit that changes,for each frame, a reference position of an area of the image data to beread out from the photoelectric conversion element by the readoutcontrol unit in the thinning readout mode, an image area selectionprocessing unit that selects an area of a region common to the pluralityof frames of the corrected image signal output from the distortioncorrection unit on the basis of the reference position of the readoutarea that is changed for each frame by the readout phase control unit inthe thinning readout mode, a plurality of storage units that temporarilystore one of the image data output from the image area selectionprocessing unit and the image data of the plurality of frames of theimage signal output from the image sensing device in one of thehorizontal averaging vertical interlaced readout mode and the verticalinterlaced readout mode, an interframe arithmetic processing unit thatgenerates new image data by executing weighted averaging for the imagedata of the plurality of frames stored in the storage units in thethinning readout mode, an interframe interpolation arithmetic processingunit that executes interframe interpolation for the image data of theplurality of frames stored in the storage units so as to mutuallyinterpolate omitted image information between the frames in one of thehorizontal averaging vertical interlaced readout mode and the horizontalfull-pixel vertical interlaced readout mode, a linear interpolation sizechange unit that executes, in accordance with the image scaling ratio,size change by linear interpolation for the image signal output from theinterframe interpolation arithmetic processing unit in one of thehorizontal averaging vertical interlaced readout mode and the horizontalfull-pixel vertical interlaced readout mode, a first selector thatselectively sends the image signal output from the image sensing deviceto one of the distortion correction unit and the plurality of storageunits in accordance with the readout scheme selected by the readoutscheme selection unit, and a second selector that selectively sends theimage signal from the storage unit to one of the interframe arithmeticprocessing unit and the interframe interpolation arithmetic processingunit.
 9. An image sensing apparatus according to claim 1, wherein thereadout scheme includes a thinning readout mode, and further comprisinga distortion correction unit that executes distortion correction for theimage data output from the image sensing device in the thinning readoutmode.
 10. An image sensing apparatus according to claim 9, wherein theimage data output from the image sensing device contains a plurality offrames, and a thinning readout rule is changed for each of the pluralityof frames in the thinning readout mode.
 11. An image sensing apparatusaccording to claim 1, wherein the readout scheme includes a thinningreadout mode, the image data output from the image sensing devicecontains a plurality of frames, and a thinning readout rule is changedfor each of the plurality of frames in the thinning readout mode.
 12. Animage sensing apparatus according to claim 11, wherein in the readoutrule that is changed for each frame, when the scaling ratio is constant,the number of pixels does not change.
 13. An image sensing apparatusaccording to claim 1, wherein the readout scheme selection unit selectsthe readout scheme on the basis of the image scaling ratio and animportant one of image quality factors including contrast, resolution,distortion, luminance moiré, and color moiré of the image to be output.14. An image data readout control method of an image sensing apparatusthat outputs an image of an object, comprising: selecting a scalingratio of the image to be output; selecting, in accordance with theselected image scaling ratio, a readout scheme of the image data to beread out from a photoelectric conversion element; and reading out theimage data from the photoelectric conversion element in accordance witha readout rule corresponding to the selected readout scheme.
 15. Animage data readout control method according to claim 14, wherein thereadout scheme includes a thinning readout mode, and distortioncorrection is executed for the readout image data in the thinningreadout mode.
 16. An image data readout control method according toclaim 15, wherein the image data readout from the photoelectricconversion element contains a plurality of frames, and a thinningreadout rule is changed for each of the plurality of frames in thethinning readout mode.
 17. An image data readout control methodaccording to claim 14, wherein the readout scheme includes a thinningreadout mode, the image data readout from the photoelectric conversionelement contains a plurality of frames, and a thinning readout rule ischanged for each of the plurality of frames in the thinning readoutmode.